This article introduces the relationship between KL divergence and cross entropy, pointing out that the expectation of information amount is $-log(p)$, and entropy is the sum of expectations. KL divergence measures the difference between two probability distributions and explains their asymmetry and asymmetry. Furthermore, the article explores the relationship between information entropy and coding, and how to use Shannon's coding theorem to calculate entropy, thereby deriving the concept of average coding length. Finally, the article discusses the application of conditional entropy and maximum likelihood estimation in cross-entropy loss functions, emphasizing another understanding of cross-entropy as a loss function....

This article discusses LLM model fine-tuning methods trained on large-scale general data sets, especially how to adapt to specific tasks through methods such as adapters, prefixes, and prompts. In particular, the low-rank adaptive (LoRA) technology proposed by Microsoft is introduced. This method reduces the amount of calculations by decomposing the update matrix into the product of two low-rank matrices, and achieves similar effects to full fine-tuning. The article also discusses the specific principles, applications and challenges of LoRA....

Adam optimization method memory usage. Generally speaking, Adam uses more information, so it requires more memory. There was an experiment. When the Optimizer selects SGD, the Batch Size can be selected to be 20, but when it is replaced with Adam, the Batch Size can only be selected to be 2. The adaptive embodiment of Adam is exponential smoothing, which alleviates drastic changes in the gradient direction and makes the convergence path smoother....

This article introduces the two main forms of the quasi-Newton method: the BFGS method and the L-BFGS method. The BFGS method uses curvature information to preprocess gradients, thereby avoiding the need for a complete calculation of the Hessian matrix in traditional methods. The L-BFGS method further optimizes this preprocessing by saving only the curvature information of the most recent m iterations to calculate an approximation of the Hessian matrix, significantly reducing memory usage and computational effort....

This article introduces the implementation of multi-head attention (MHA) for handwriting. The article first defines the MultiHeadAttention class, which is used to process multi-dimensional input data and generate output. During the implementation process, the author describes in detail how to build the attention mechanism through linear transformation, Scaled Dot-Product Attention and Softmax operations. Experimental results show that using einsum notation can simplify code writing and improve readability....

This article explores the causes and mitigation methods of vanishing/exploding gradients in neural networks. First of all, the article points out that the gradient descent method will encounter the problem of gradient disappearance or explosion when dealing with complex networks, which is mainly caused by chain derivation and gradient explosion/disappearance. Next, the article lists several common improvement methods: pre-training plus fine-tuning, gradient clipping and regularization, using batch normalization (batchnorm), residual structure, and LSTM, etc. These methods aim to optimize the training process of neural networks, reduce the risk of vanishing or exploding gradients, and improve model performance....